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(57) During the operation of a document processing 
system machine and job data are collected from a doc- 
ument processing system. Optionally, machine informa- 
tion, both for the specific machine and population based 
data are acquired from a database or server. Having ac- 
cumulated the job data and machine data, the diagnos- 
tic inference engine performs an analysis to determine 
the initial diagnosis of the document processing system. 
After obtaining the initial diagnosis, the system deter- 
mines the test patterns to be printed and the image qual- 
ity tests to be performed. The system then prints test 
patterns, and scans the patterns to determine image 
quality parameters and/or to automatically identify im- 
age defects. Optionally, image defect information may 
also be provided by the customer or the service engi- 
neer via a user interface. Next, a diagnostic inference 
engine uses the results of the image quality analysis to 
refine the initial diagnosis. Then, the diagnostic results 
are output, stored, and optionally displayed to, for ex- 
ample a customer or a customer service engineer. 
Based on the diagnostic results, and if problems are 
found, the machine enters a repair sequence. In partic- 
ular, the machine can request either a customer or a 
customer service engineer repair action, or, alternative- 
ly, enter an auto-correction or an auto-calibration mode 
to repair itself. Upon completion of one or more, or any 
combination, of these repair actions, the machine veri- 
fies its operation and again checks to ensure the repairs 
have been completed successfully. 
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Description 

[0001] This invention relates to failure diagnosis, pre- 
diction and remediation of document processing sys- 
tems based on an image quality defect analysis. 
[0002] It is well known that customer satisfaction can 
be improved and maintenance costs reduced if prob- 
lems with copiers and printers can be fixed before they 
become serious enough to warrant a service call by the 
customer. Systems exist that enable printers and copi- 
ers to call for service automatically when sensors detect 
certain operating parameters outside of permissible 
ranges. Generally, these systems take effect after a 
threshold has been reached where the degradation in 
image quality is directly observable by the customer. For 
example, given the large number of operating parame- 
ters that need to be tracked during operation, a given 
parameter at a certain value may or may not be a prob- 
lem depending on the cumulative values of the other pa- 
rameters in the system. Additionally, systems exist that 
attempt to diagnose failures in document processing 
systems based on image quality analysis of the print. 
Systems exist for image quality based diagnosis of cop- 
iers over telephone lines such as U.S. Patents, 365, 31 0, 
incorporated herein by reference in its entirety. 
[0003] The systems and methods of this invention 
provide for automated diagnosis, prediction and reme- 
diation of failures in document processing systems 
based on an image quality defect analysis in conjunction 
with a machine/device data analysis. Specifically, the 
systems and methods of this invention automatically 
identify image quality problems in document processing 
systems, such as analog and digital copiers, printers, 
scanners, facsimiles, and the like based on all direct and 
indirect marking technologies including xerography, 
inkjet, liquid ink, lithography, or the like, by analyzing 
specific test patterns via techniques such as image 
processing and pattern recognition. 
[0004] Secondly, the systems and methods of this in- 
vention, in a diagnostic expert system, automatically di- 
agnose and/or predict machine failures based on the 
combined results of the image quality analysis and ma- 
chine data analysis. Thirdly, this invention separately 
provides systems and methods for determining and ex- 
ecuting an appropriate action based on the results of the 
diagnostic/prediction analysis. Such actions could in- 
clude, for example, guiding the customer through a re- 
pair procedure, automatic scheduling of service, parts 
and/or consumables and automated remediation of 
faults, for example, as discussed in copending EP Ap- 
plication of even date, Attorney Docket No. 
RSJ06866EP, and incorporated herein by reference in 
its entirety. 

[0005] Additionally, this invention separately provides 
systems and methods for the presentation of the results 
of the failure prediction, diagnosis or remediation, local- 
ly, or, remotely, such as, for example, on a computer us- 
er interface, via e-mail, via a web page, via a paging 



service or cellular phone, or the like, and for the storage 
of the results, for example, in one or more databases, 
servers, or device history logs. Therefore, the systems 
and methods of this invention are capable of performing 

5 intelligent self-diagnosis, correction and notification of 
problems that lead to image quality defects. 
[0006] The systems and methods of this invention 
provide a diagnostic/prognostic analysis device for doc- 
ument processing systems. 

w [0007] This invention separately provides systems 
and methods for identifying image quality defects. 
[0008] This invention also separately provides sys- 
tems and methods for automatically diagnosing and/or 
predicting machine failures based on a combined anal- 

15 ysis of image quality data and device data. This inven- 
tion additionally provides systems and methods that 
provide a unified framework to incorporate a variety of 
different diagnostic technologies. 
[0009] This invention separately provides systems 

20 and methods for determining an appropriate action 
based on the results of the diagnostic/prognostic anal- 
ysis. 

[0010] This invention separately provides systems 
and methods that allow an automatic scheduling of serv- 
es ice, parts and/or consumables to be provided to an elec- 
tronic system. 

[0011] This invention separately provides systems 
and methods that guide acustomer through a repair pro- 
cedure. 

30 [0012] This invention separately provides systems 
and methods that allow an automated remediation of 
faults, either completely or partially, and with or without 
human intervention. 

[0013] This invention separately provides systems 
35 and methods for the presentation of the results of the 
failure prediction, diagnosis or remediation either locally 
or remotely. 

[0014] The invention separately provides systems 
and methods that allow electronic systems to be inter- 
ne rogated and controlled remotely over a network for the 
acquisition of data for use in failure prediction, diagnosis 
and/or remediation. 

[0015] This invention additionally provides systems 
and methods for pooling data related to a plurality of 
45 document processing systems to accomplish failure 
prediction, diagnosis and remediation of the distributed 
electronic systems. 

[0016] Specifically, at any point during the operation 
of a document processing system, the diagnostic/prog- 

50 nostic systems and methods of this invention can be in- 
voked. At this time, machine and job data are collected 
from the document processing system. The machine in- 
formation can be obtained from the machine that is be- 
ing diagnosed and/or from one or more knowledge serv- 

55 ers or databases that contain machine specific informa- 
tion as well as population information from a fleet of sim- 
ilar machines via a network connection. Having accu- 
mulated the job data and machine data, the diagnostic 
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inference engine performs an analysis to determine the 
initial diagnosis of the document processing system. Af- 
terobtaining the initial diagnosis, the system determines 
the test patterns to be analyzed and the image quality 
tests to be run. Then, system prints test patterns, and 
scans the patterns to identify image quality defects. 
Next, a diagnostic inference engine analyzes the image 
defects in the printed test patterns to refine the initial 
diagnosis. Then, the diagnostic results are output, and 
optionally displayed to, for example, a customer or cus- 
tomer service engineer. 

[0017] Based on the diagnostic results, the machine 
enters a repair sequence if problems are found. In par- 
ticular, the machine can request either a customer or a 
customer service engineer repair action, or, alternative- 
ly, enter an auto-correction or auto-calibration mode to 
repair itself. Upon completion of one or more, or any 
combination, of these repair actions, the machine veri- 
fies its operation and again checks to ensure the repairs 
have been completed successfully. 
[0018] It is to be noted that at any time during the di- 
agnostic or remediation process, additional data can be 
obtained from the machine by interrogating or control- 
ling the machine and/or requesting additional test pat- 
terns be printed, scanned and analyzed. 
[0019] These and other features and advantages of 
this invention are described in or are apparent from the 
following detailed description of the preferred embodi- 
ments. 

[0020] The preferred embodiments of the invention 
will be described in detail, with reference to the following 
figures, wherein: 

Fig. 1 is afunctional block diagram illustrating a first 
embodiment of the diagnostic system according to 
this invention; 

Fig. 2 is a functional block diagram showing a sec- 
ond embodiment of a diagnostic system according 
to this invention; 

Fig. 3 is a functional block diagram showing a third 
embodiment of a diagnostic system according to 
this invention; 

Fig. 4 is afunctional block diagram showing a fourth 
embodiment of a diagnostic system according to 
this invention; 

Fig. 5 is a functional block diagram showing a fifth 
embodiment of a diagnostic system according to 
this invention; 

Fig. 6 is a functional block diagram showing a sixth 
embodiment of a diagnostic system according to 
this invention; 

Fig. 7 is a functional block diagram showing a sev- 
enth embodiment of a diagnostic system according 
to this invention; 

Fig. 8 is a workflow diagram illustrating an exem- 
plary system workflow according to this invention; 
and 

Fig. 9 is a flowchart outlining one exemplary em- 



bodiment of a method for diagnosing system faults 
in accordance with this invention. 

[0021] In a document processing system, such as a 
5 printer, copier, scanner, facsimile machine, or a multi- 
function device incorporating one or more of the above, 
when the user is not satisfied with the machine's per- 
formance, or when preventative maintenance is de- 
sired, the diagnostic systems of this invention are in- 
fo voked. In particular, the diagnosis routine can be initiat- 
ed by the machine itself or can be initiated by a user or 
other operator, or by a diagnostic server residing on a 
distributed network. The first step to diagnosing/predict- 
ing a system problem is to collect and analyze relevant 
15 machine data. Such data includes, but is not limited to, 
control data such as commands issued by system and 
subsystem controllers, scheduling and timing data, set- 
point and actuator data, sensor data, state estimate data 
and so on; diagnostic data such as fault counts, error 
20 counts, event counts, warning and interlock counts, cal- 
ibration data, device set-up data, high frequency service 
item information, service history data, machine history 
data and so on, virtual sensor readings, or job related 
data such as copy counts, percentage area coverage, 
25 documentsize and type and the like. The data could also 
include environmental conditions such as temperature 
and humidity of the operating environment, and ma- 
chine usage information such as, for example, the type 
and frequency of machine usage, and machine config- 
30 uration information. This data collection can occur in real 
time, for example, as the user is processing documents, 
or, during a special diagnostic "stress mode" or test 
mode, where the document processing system is exer- 
cised appropriately to extract useful diagnostic informa- 
35 tion that may otherwise be unavailable in the normal ma- 
chine operation mode. 

[0022] Alternatively, the data collection can occur on 
an ongoing basis before service of the machine is at- 
tempted, or even when the machine is in a limp-along 
40 mode. The information may be obtained from the ma- 
chine itself, or from one or more servers connected to 
the machine in a distributed network that store informa- 
tion pertinent to the machine being diagnosed. In addi- 
tion to the machine specific information, fleet-wide ma- 
45 chine data resident in population databases or servers 
may also be used in the diagnostic/prognostic analysis. 
This data is then forwarded to a diagnostic inference en- 
gine that analyzes the data and provides an initial diag- 
nosis. If this diagnosis proves to be incomplete, the im- 
50 age quality analysis module is activated. Then, based 
on the results of the initial diagnosis, a determination is 
made as to what test prints are to be scanned and what 
image quality tests are to be performed. A set of pre- 
defined test patterns are then printed and subsequently 
55 scanned, or scanned directly, depending on whether the 
test patterns are stored internal to the machine being 
diagnosed, are available on a networked drive, or, are 
available in hardcopy form. This scanning can be ac- 
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complished by using an external scanner, or by using a 
scanner incorporated into the document processing 
system. Furthermore, the scanning process can be user 
mediated, or can be completely automated if the device 
includes a media transport mechanism to re-circulate a 
document to the device's scan path from the output 
path, such as, for example, from the output of the fuser 
module in a printer. The directly scanned image, or the 
printed and subsequently scanned image, is then ana- 
lyzed by the image quality analysis module to detect and 
identify the presence of defects in the image and to de- 
termine image quality parameters. The results of the im- 
age quality/defect analysis are then fed to the diagnostic 
engine which refines its original diagnosis based on the 
new information received from the printed test pattern 
(s), and produces a final list of failures, suspected fail- 
ures, or impending failures. By collecting and analyzing 
the machine data first, and using the results of that anal- 
ysis to direct the extent and content of the image quality 
analysis, the number of test patterns that need to be cre- 
ated and hence, the amount of time spent by the cus- 
tomer, a customer service engineer, a third party repair 
person or an autonomous repair agent in diagnosing 
and servicing the machine can be reduced. The ma- 
chine data analysis and image quality analysis can take 
place in any order, or even in parallel, depending on the 
specific system implementation. Furthermore, at any 
time during the diagnostic process, the machine may be 
queried or controlled to obtain additional data from the 
machine such as, for example, by running additional 
tests. Likewise, additional test patterns may also be 
printed, scanned and analyzed at any time during the 
process, if necessary. Additionally, the extent of the di- 
agnosis process can limited, for example, based on a 
predefined user authorization scheme and correspond- 
ing user identification. 

[0023] The results from the diagnostic engine are then 
conveyed to, for example, the customer or the customer 
service engineer. Where possible, the system attempts 
to repair itself. Otherwise, either the customer or a serv- 
ice engineer is contacted to undertake the appropriate 
repair action. In addition, a request could be forwarded 
automatically to a parts/consumables supplier. Once the 
machine has been repaired, a verification process is in- 
itiated to ensure the repairs were successful. Finally, the 
results of the diagnostic/prognostic analysis and the re- 
pair procedures are logged into a machine service log 
or one or more databases. 

[0024] Fig. 1 illustrates one exemplary embodiment 
of the diagnostic system 10. The diagnostic system 10 
comprises a document processing system 1 00, such as 
a xerographic marking engine, an ink jet printer, a litho- 
graphic printer, a color or black and white printer, a color 
or black and white photocopier, a color or black and 
white scanner, a facsimile machine, or the like, a scan- 
ner 110, an image quality analysis circuit 120, a job con- 
trol circuit 130, a system/subsystem controller(s) 140, 
one or more sensors 150, a memory 160, one or more 



virtual sensors 170, a diagnostic pre-processor circuit 
180, a diagnostic inference engine circuit 190, a diag- 
nostic controller 200, a knowledge server/database 230, 
a local user interface 210 and a remote user interface 

5 220, all interconnected by links 25. 

[0025] It is to be appreciated that any one of, or a por- 
tion of the components of the diagnostic system can be 
located anywhere including on the actual document 
processing system itself, on a distributed network, or a 

10 collocated or remote diagnostics location. 

[0026] Furthermore, the links 25 can be wired or wire- 
less links or any other known or later developed element 
(s) that is capable of supplying electronic data to and 
from the connected elements. Furthermore, links 25 can 

15 be any one of, or combination of, a direct serial connec- 
tion, a distributed network such as an intranet, a local 
area network, a metropolitan area network, a wide area 
network, a satellite communication network, an infrared 
communication network, the Internet, or the like. 

20 [0027] Since sensors are typically a scarce commod- 
ity in document processing systems, in order to diag- 
nose machine failures down to the individual compo- 
nent, an easily replaceable unit, or a customer replace- 
able unit, typically more machine information than that 

25 provided by the on-board sensors is necessary. To this 
end, the virtual sensor is used to augment the informa- 
tion provided by the real sensors on the machine. The 
virtual sensors 170 are based on quantitative analysis 
of machine signals, such as, for example, ground return 

30 line currents, in order to extract useful diagnostic infor- 
mation from the signals. Alternatively, they could be 
counters that keep track of signal changes, fault counts, 
calibration runs, pages processed, etc. Depending on 
the document processing system, the nature of the fail- 

35 ures to be diagnosed, and the nature of the available 
information, a variety of different techniques of varying 
complexity can be used for the virtual sensors 1 70. Such 
techniques, such as those described in U.S. Application 
No. 60/154,016, incorporated herein by reference in its 

40 entirety, include but are not limited to, threshold analy- 
sis, statistical analysis, such as means and variance 
analysis, signature analysis, pattern analysis, trend 
analysis, event sequence analysis, timing analysis, rate 
analysis, counting mechanisms, such as, for example, 

45 signal change counters, state estimation analysis, and 
the like. 

[0028] In operation, data is collected from one or more 
of the job control circuit 1 30, the system/subsystem con- 
trollers 140, the one or more sensors 150, the one or 

50 more virtual sensors 1 70, and the memory 1 60. This da- 
ta is derived from the operational characteristics of the 
document processing system 100 and forwarded to the 
diagnostic pre-processing circuit 180. In addition, both 
the individual machine information and the fleet-wide 

55 population statistics may be obtained from the knowl- 
edge server/database 230. The pre-processing circuit 
1 80 collects the machine and job data from the various 
sensors, controllers, or the like, and translates this data 
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into a format recognizable by the diagnostic inference 
engine circuit 190. For example, this pre-processing 
may involve discretization of analog data received from 
one or more sensors into qualitative values. Alternative- 
ly, it may involve translating machine signals into dis- 
crete event sequences as described in U.S. Application 
No. 60/1 54,01 6, that can be recognized by the diagnos- 
tic inference engine circuit 190. The diagnostic infer- 
ence engine circuit 190 performs the task of reasoning 
with the multiple sources of diagnostic/prognostic infor- 
mation and isolating the root of the cause for the failure, 
actual or impending. The diagnostic inference engine 
circuit 1 90 can be based on one or more of the following 
diagnostic/prognostic technologies, for example, model 
based diagnosis, discrete event systems diagnosis, 
b ayes i an networks, causal networks, neural networks, 
artificial intelligence, rule-based systems, expert sys- 
tems, fuzzy logic analysis, and/or look-up tables. There- 
fore, based on the analysis of the data output by the pre- 
processor circuit 180, the diagnostic inference engine 
circuit 190 determines the list of components that have 
failed, are suspected to have failed or estimated to fail. 
[0029] The diagnostic inference engine circuit 190 
then determines an initial diagnosis of the document 
processing system 1 00 based on the data received from 
the various sensors, the virtual sensors, the controllers, 
and the memory units via the pre-processor circuit 1 80. 
Based on this initial diagnosis, and in cooperation with 
the diagnostic controller 200, the initial diagnosis is for- 
warded to one or more of the local user interface 21 0 or 
the remote user interface 220, and displayed. Addition- 
ally, the diagnostic controller 200 initiates a print and 
scan sequence to determine image quality parameters 
and/or detect image defects. In particular, the diagnostic 
controller decides, based on the initial diagnosis, what 
test prints are to be scanned and what image quality pa- 
rameters are to be determined and/or what defects are 
to be recognized and characterized. The document 
processing system 100 is then be directed to produce 
one or more test pattern prints, stored either in its inter- 
nal memory, or on a network drive in a distributed net- 
work environment. The test pattern prints are then for- 
warded to the scanner 1 1 0 for digitizing. Alternately, the 
customer, the customer service engineer, or the like, can 
directly scan the hard copy test prints. 
[0030] It is to be appreciated that the scanner 1 1 0 can 
be collocated with the document processing system 1 00 
or, alternatively, be a stand-alone device or incorporated 
into, for example, a dedicated scanner and diagnostic 
system. Also, the printing and scanning process can be 
completely automated if a transport mechanism is avail- 
able to re-circulate the printed test pattern to the scan- 
ner, for example, in the case where the scanner is in- 
corporated into the document processing system being 
diagnosed. 

[0031] The digitized images of the test patterns pro- 
duced by scanner 1 1 0 are forwarded to the image qual- 
ity analysis circuit 1 20. The image quality analysis circuit 



120 analyzes the image, using commonly known image 
processing techniques such as, for example, Fourier 
transform analysis, histogramming, edge detection, 1 -D 
projections, segmentation, classification, artifact detec- 

5 tion, FIR filtering, morphological analysis, wavelet anal- 
ysis and statistical analysis pattern recognition tech- 
niques, neural networks, or the like, to evaluate image 
quality parameters and/or identify defects in the image 
(s). Furthermore, the image processing can compen- 

10 sate via a calibration circuit for any defects introduced 
by the scanner., and then determine the image quality 
parameters and/or detect the presence of any defects 
in the printed document. 

[0032] Specifically, the image quality defect recogni- 

15 tion circuit 120 detects the presence of one or more of 
a set of pre-defined defects in the input scanned image 
and/or calculates the values of one or more image qual- 
ity parameters. Such defects and parameters include, 
but are not limited to, lines, bands, streaks, spots, mot- 

20 tie, halo, splatter, light images, dark images, blank im- 
ages, deletions, background ink contamination, graini- 
ness, ghosting, skew, registration defects, color bal- 
ance, color registration, color variation within a page, 
print-to-print variation, inboard-outboard variation, loss 

25 of a tone-reproduction, line width, edge width, line den- 
sity, line edge, line shape, sharpness, contours, reload, 
pixel to pixel placement defects, and the like. In partic- 
ular, the image quality analysis and defect recognition 
are accomplished through the use of image processing 

30 and/or pattern recognition algorithms such as those de- 
scribed in copending US Patent Applications 
09/450,185, 09/450,183, 09/450,182, 09/450,181, 
09/450,180, 09/464,596 and 09/450,177, incorporated 
herein by reference in their entirety. 

35 [0033] In addition to the image processing algorithms, 
the image quality defect recognition 120 may also have 
an optical character recognition component , bar code 
or glyph recognition component that ensures that the 
correct test patterns have been scanned and scanned 

40 in the right sequence and orientation. If the correct test 
pattern was not scanned, in cooperation with the diag- 
nostic controller 200, the user can be re-directed to re- 
print and re-scan the right prints. 
[0034] Then, upon completion of the image quality de- 

45 feet analysis, the image quality defect recognition circuit 
1 20 outputs a set of metrics that qualify and quantify the 
one or more defects, and/or image quality parameters. 
In addition to automated recognition of the image quality 
defects, input may also be obtained from the customer, 

50 and/or the customer service engineer via the user inter- 
face 21 0 or 220, for characterizing the defects observed 
either in the test prints, or in the customer prints. This 
additional input from the user can be used to augment 
and/or verify the results of the image quality analysis 

55 module, and/or to reduce the number of test prints to be 
scanned and the number of defects to be characterized, 
similar to the manner in which the results of the initial 
diagnosis based on machine data are used. 
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[0035] The results of the defect recognition are for- 
warded to the diagnostic pre-processor 1 80, via link 25. 
The pre-processing in this case may involve compari- 
sons of the results of the image quality analysis from 
two or more scanned images to check, for example, the 
presence of defects in all prints, the registration of a de- 
fect with respect to a page, the effects of rotating/mag- 
nifying a test pattern, and so on. The pre-processor cir- 
cuit 1 80, now having information pertaining to the image 
quality, forwards this additional information to the diag- 
nostic inference engine circuit 190. The diagnostic in- 
ference engine circuit 190 then refines the initial diag- 
nosis and outputs the results of this diagnosis to one or 
more of the local user interface 210 and/or the remote 
user interface 220. For example, the diagnostic infer- 
ence engine circuit 1 90 can forward to either one of the 
user interfaces information indicating that the machine 
is operating acceptably, or that one or more failures are 
predicted to occur, or have occurred. 
[0036] The diagnostic inference engine circuit 190 
outputs information pertaining to the operational status 
of the document processing system. In particular, the 
list includes, but is not limited to, status information 
about the system, a list of one or more failed compo- 
nents, information pertaining to components that are 
predicted to fail, or the like. These results are output and 
displayed locally on the user interface of the system and/ 
or a local diagnostic server on the customer side and/or 
transmitted remotely to any entity needing the informa- 
tion, such as a customer service engineer, a diagnostic 
server, a part supplier, or the like. Additionally, it is to be 
appreciated that the results of the diagnostic inference 
engine circuit 190 need not be limited to visual notifica- 
tions, but can also include audio notifications or any 
means obvious to one of ordinary skill in the art to notify 
the appropriate party. Examples of such notification 
methods include notifications via e-mail, paging, cellular 
phones, a web page, or the like. Finally, the results of 
the diagnostic/prognostic analysis may also be stored 
in machine history logs and/or in one or more databases 
or servers. 

[0037] If the machine is operating correctly, the diag- 
nostic routine is complete. However, if there are any er- 
rors, or predicted errors determined in the refined diag- 
nosis step, the diagnostic system 10 performs one or 
more of contacting a customer or customer service en- 
gineer for commencement of repair action. Alternatively, 
the diagnostic system 1 0 communicates with the docu- 
ment processing system 100 to invoke any appropriate 
auto-repair procedure such as machine set-up proce- 
dures, color balancing, color calibration, placing the ma- 
chine in a low productivity mode, etc. The results of the 
automated repair procedures, such as, for example, the 
machine parameters changed during a set-up proce- 
dure, are displayed on the user interface of the system, 
and/or a diagnostic server and/or transmitted remotely 
to a customer service engineer and, in addition, stored 
in the machine history logs. Upon completion of the cus- 



tomer or customer service engineer repair action, and/ 
or an automatic repair sequence, the machine's opera- 
tion is verified and, if correct, the diagnostic routine com- 
pleted. In the case of customer repair, the customer is 
5 guided through a repair procedure. In addition, the sys- 
tem may be equipped with a security system that allows 
different classes of customers, such as the end-user, 
key operator, and system administrator different levels 
of access to perform repair actions. For example, the 
10 end-user may have access to replace consumables, but 
no access to run a color calibration routine, or adjust 
machine setpoints. The key operator may have access 
to run a calibration routine and replace consumables, 
but denied access to change set-point values in the sys- 
15 terns internal memory. All users may on the other hand, 
have permissions to place a service call. 
[0038] It is to be noted that in the above embodiment 
the machine data analysis precedes the image quality 
analysis. However, the machine data analysis and the 
20 image quality analysis can take place in any order, or 
even in parallel, depending on the specific system im- 
plementation. It is to also to be noted that at any time 
during the diagnostic process, the machine may be que- 
ried or controlled to obtain additional data from the ma- 
25 chine such as, for example, by running additional tests. 
Likewise, additional test patterns may also be scanned 
and analyzed at any time during the process if neces- 
sary. 

[0039] Figs. 2-7 illustrate various exemplary embodi- 
30 ments of the locations of the components of the diag- 
nostic/prognostic system 10 based on various consid- 
erations such as cost, network bandwidth, processor re- 
quirements, memory requirements, latency, data trans- 
mission rates, firewall and security issues, marketing 
35 considerations, or the like. 

[0040] In particular, Fig. 2 illustrates an exemplary 
embodiment where most of the diagnostics are per- 
formed in the document processing system itself, how- 
ever, the display and notification information can be 
40 communicated anywhere. For example, the copier 300 
includes the image quality analysis circuit 120 and the 
diagnostic inference engine circuit 1 90. Additionally, the 
printer 400 comprises the image quality analysis circuit 
1 20 and the diagnostic inference engine circuit 1 90. The 
45 printer 400 communicates with the scanner 1 1 0 to digi- 
tize one or more test patterns created by the printer 400. 
Where necessary to use an external scanner, the copier 
300 may also communicate with the scanner 110. The 
local diagnostic server 500 is located on the copier 300 
50 and printer 400 side of the firewall 50 while, in this ex- 
emplary embodiment, the diagnostic server 700 is locat- 
ed on the opposite side of the firewall 50, and the port- 
able work station 600 may be on either side of the fire- 
wall 50. 

55 [0041] Fig. 3 illustrates a second exemplary embodi- 
ment where the diagnostic computation is carried out on 
a local diagnostic server at, for example, the customer 
site. In this case, the image quality analysis circuit 120 
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is located in the copier 31 0 or in the printer 41 0, which 
is connected to the scanner 110, and the diagnostic in- 
ference engine circuit 1 90 is located within the local di- 
agnostic server 510. By having the image quality anal- 
ysis circuit 120 located on the printer/copier, the 
scanned image data, which may be of high bandwidth, 
does not need to be transported across the customer's 
network. Therefore, the remainder of the information 
needed by the diagnostic engine, i.e., all the relevant 
machine data, should be transmitted to the local server, 
either in real-time, or stored in memory, such as the 
memory 160, and then transmitted to the server. As in 
the case of Figure 2, the copier 310 may also commu- 
nicate with the scanner 1 1 0 where it is necessary to use 
an external scanner. The local diagnostic server 510 is 
located on the copier 310 and printer 410 side of the 
firewall 50 while, in this exemplary embodiment, the di- 
agnostic server 710 is located on the opposite side of 
the firewall 50, and the portable work station 610 may 
be on either side of the firewall 50. 
[0042] In Fig. 4, the local diagnostic server houses the 
image quality analysis circuit 120 and the diagnostic in- 
ference engine circuit 1 90. In this case, the scanner 1 1 0 
is connected directly to the local diagnostic server 520. 
In this instance, the copier 320 and printer 420 do not 
house any of the diagnostic algorithms. Again, the cop- 
ier 320 may be connected to an external scanner 110, 
if necessary. The local diagnostic server 520 is located 
on the copier 320 and printer 420 side of the firewall 50 
while, in this exemplary embodiment, the diagnostic 
server 720 is located on the opposite side of the firewall 
50, and the portable work station 620 may be on either 
side of the firewall 50. 

[0043] Fig. 5 illustrates another exemplary embodi- 
ment in which the diagnostic inference engine circuit 
1 90 is located in the portable workstation 630, that could 
be on either side of the firewall 50. The copier 330 and 
the printer 430, which additionally are, or can be con- 
nected to the scanner, comprise the image quality anal- 
ysis circuit 120. Alternately, the image quality analysis 
circuit 1 20 may reside in the local diagnostic server 530. 
The local diagnostic server 530 is located on the copier 
330 and printer 430 side of the firewall 50 while, in this 
exemplary embodiment, the diagnostic server 730 is lo- 
cated on the opposite side of the firewall 50, and the 
portable work station 630 may be on either side of the 
firewall 50. 

[0044] Fig. 6 illustrates another exemplary embodi- 
ment where the image quality analysis is on the docu- 
ment processing system or on the local server, and the 
diagnostic inference engine is part of a diagnostic server 
located on the opposite side of the firewall 50. In partic- 
ular, the copier 340 includes an image quality analysis 
circuit 1 20, and the printer 440 includes the image qual- 
ity analysis circuit 120. The printer, and optionally the 
copier, are connected to the scanner 110. The local di- 
agnostic server 540 includes an image quality analysis 
circuit 120, and the diagnostic server 740 includes a di- 



agnostic inference engine circuit 1 90. While in this par- 
ticular embodiment, the image quality analysis circuit is 
located on the document processing system or on the 
local server, it can be appreciated that the circuit can 

5 also reside on the diagnostic server along with the diag- 
nostic inference engine .As with the above described 
embodiments, the portable workstation 640 can be on 
either side of the firewall 50, but in this case, does not 
include the diagnostic inference engine. 

w [0045] Figs. 7 illustrate an exemplary embodiment 
where both the image quality analysis circuit and the di- 
agnostic inference engine can be implemented in a sep- 
arate "image quality diagnostic tester." This image qual- 
ity diagnostic tester may also include an scanner. The 

15 diagnostic tester can be connected to the document 
processing system via a local connection medium, such 
as a serial link, the ethernet or any other well known or 
later developed wired or wireless link. Additionally, the 
diagnostic tester can be connected to the portable work- 

20 station 650 and/or the diagnostic server 750 outside the 
firewall 50. In particular, this embodiment includes cop- 
ier 350, printer 450, the diagnostic tester 555, which in- 
cludes the diagnostic inference engine circuit 190, the 
image analysis circuit 1 20, scanner 1 1 0, and, optionally, 

25 a local diagnostic server 550, the firewall 50 and the di- 
agnostic server 750. 

[0046] Fig. 8 illustrates an exemplary workflow dia- 
gram showing data flow between the various compo- 
nents of the diagnostic/prognostic system. In particular, 

30 control begins in step S1 where unsatisfactory image 
quality has been detected. Next, in step S2, the image 
quality diagnosis is invoked. Then, in step S4, the ma- 
chine data service is invoked to acquire machine data 
as well as optionally perform additional tests to derive 

35 additional machine data. 

[0047] In step S6, the machine tests are run. Next, in 
step S8, the acquired data is preprocessed and stored 
in a database. Then, in step S10, the first phase diag- 
nosis is run on the machine data and an initial diagnosis/ 

40 prognosis is obtained. Then, in step S12, a determina- 
tion is made as to which image quality tests should be 
performed and which test patterns are to be scanned. 
In step S14 the determined test patterns are retrieved, 
printed and scanned. The scanned images are then ac- 

45 quired in step S16. Then, in step S20, the scanned im- 
age data is forwarded to the image quality analysis cir- 
cuit for image quality analysis and defect recognition. 
[0048] In step S22, the results of the image quality 
analysis, i.e, the image quality defect data and the im- 

50 age quality parameter data, are preprocessed. Next, in 
step S24, the phase 2 diagnosis/prognosis is run and 
the list of failures that have occurred, or are impending, 
are determined. Then, in step S26, the repair action is 
planned and recommended repair actions forwarded to 

55 the database in conjunction with all of the final and the 
intermediate results of the diagnostic/prognostic analy- 
sis. In addition, the scanned image data, the performed 
image quality tests and the test patterns used, may all 
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be stored in the database. 

[0049] In step S28, and depending on the recom- 
mended repair action, an autonomous repair sequence 
is initiated. The data pertaining to the automated repair 
action, such as, for example, the setpoint changes, are 
written to the database. Alternatively, in step S30, where 
a customer repair action is recommended, the neces- 
sary or appropriate information is sent to the customer. 
Next, the customer repairs the machine. Alternatively 
again, in step S32, a call for service is made based on 
the recommended repair action. This includes forward- 
ing the call to a service monitoring system which dis- 
patches a service call in step S34 to a customer service 
engineer and/or forwards a service call in step S36 to a 
portable workstation. Then, in step S38, the customer 
service engineer repairs the machine. In step S40, the 
repair action and the satisfactory operation of the ma- 
chine are verified and the diagnostics/prognostics se- 
quence ends. 

[0050] Fig. 9 illustrates an exemplary embodiment of 
one method for diagnosing/predicting document 
processing system failures in accordance with this in- 
vention. Control begins in step S100 and continues to 
step S1 1 0. In step S1 1 0, the diagnostic routine is initiat- 
ed. Next, in step S120, machine data and job data are 
collected and stored. Then, in step S1 30, the diagnostic 
inference engine operates on one or more of the ma- 
chine and job data to obtain an initial diagnostics. Con- 
trol then continues to step S140. 
[0051] In step 140, the test patterns and the corre- 
sponding image quality analysis tests to be performed 
are determined. Next, in step 1 50, test patterns are print- 
ed and scanned. Then, in step S1 60, a determination is 
made as to whether the correct test patterns have been 
scanned, and whether they have been scanned in the 
correct sequence. If the correct test patterns have not 
scanned, control continues to step S1 70 where the user 
is requested to scan the prints correctly. Control then 
returns to step S150. Otherwise, control jumps to step 
S180. 

[0052] In step S1 80, the image quality analysis is ini- 
tiated to identify image defects. Next, in step S190, the 
diagnostic inference engine refines this initial diagnosis 
based on one or more of the found image defects and 
the machine and job data. Then, in step S200, the re- 
sults of the diagnosis are output and/or stored. Control 
then continues to step S210. 

[0053] In step S210, a determination is made whether 
the machine is operating correctly. If machine failures 
were found to have occurred, control continues to step 
S220. In steps S220 and S230, the appropriate repair 
action is determined and one or more of a customer, a 
customer service engineer, or an automatic machine re- 
pair action are commenced. Then, in step S240, the op- 
eration of the machine is verified. Control then returns 
to step S210. 

[0054] Otherwise, control jumps to step S250 where 
the diagnostic routine ends. Control then continues to 



step S260 where the control sequence ends. 
[0055] As shown in Figs. 1-8, the diagnostic system 
is preferably implemented either on a single program 
general purpose computer or separate program general 

5 purpose computer. However, the diagnostic system can 
also be implemented on a special purpose computer, a 
programmed microprocessor or microcontroller and pe- 
ripheral integrated circuit element, an ASIC, or other in- 
tegrated circuit, a digital signal processor, a hard-wired 

10 electronic or logic circuit such as a discreet element cir- 
cuit, a programmable logic device such as a PLD, PLA, 
FPGA, PAL, or the like. In general, any device capable 
of implementing a finite state machine that is in turn ca- 
pable of implementing the a flowcharts shown in Figs. 

15 8-9 can be used to implement the diagnostic system. 
[0056] Furthermore, the disclosed method may be 
readily implemented in software using object or object- 
oriented software development environments that pro- 
vide portable source code that can be used on a variety 

20 of computer, workstation and/or personal digital assist- 
ant hardware platforms. Alternatively, the disclosed di- 
agnostic system may be implemented partially or fully 
in a hardware using standard logic circuits or a VLSI de- 
sign. Whether software or hardware is used to imple- 

25 ment the systems in accordance with this invention is 
dependent on the speed and/or efficiency requirements 
of the system, the particular function, and the particular 
software or hardware systems or microprocessor or mi- 
crocomputer systems being utilized. The diagnostic sys- 

30 terns and methods described above, however, can also 
be readily implemented in hardware or software using 
any known or later-developed systems or structures, de- 
vices and/or software by those skilled in the applicable 
art without undue experimentation from the functional 

35 description provided herein together with a general 
knowledge of the computer arts. 
[0057] Moreover, the disclosed methods may be 
readily implemented as software executed on a pro- 
grammed general purpose computer, a special purpose 

40 computer, a microprocessor, or the like. In this case, the 
methods and systems of this invention can be imple- 
mented as a routine embedded on a personal computer 
such as a Java® or CGI script, as a resource residing 
on a server or graphics workstation, as a routine em- 

45 bedded in a diagnostic system, personal digital assist- 
ant interface, dedicated hand-held device, or the like. 
The diagnostic system can also be implemented by 
physically incorporating the system and method into a 
software and/or a hardware system, such as the hard- 

50 ware and software systems of a workstation or dedicat- 
ed diagnostic system. 

[0058] It is, therefore, apparent that there has been 
provided, in accordance with the present invention, sys- 
tems and methods for an automated image quality 
55 based diagnostic system for document processing sys- 
tems. 
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Claims 



A system for automated image quality based prog- 
nostics, diagnostics and remediation of at least one 
document processing system comprising: 

a data acquisition and diagnostic processing 
circuit that collects and processes data; 
a diagnostic inference engine circuit that ana- 
lyzes the data to obtain an initial diagnosis; and 
an image quality analysis circuit that identifies 
at least one defect in data corresponding to at 
least one image, wherein the diagnostic infer- 
ence engine circuit refines the initial diagnosis 
based on the identified defects. 



10 



15 



8. The system of any of the preceding claims, wherein 
one or more of the data and a fleet-wide population 
statistic data is obtained from at least one of a 
knowledge server database and a memory. 

9. The system of any of the preceding claims, further 
comprising: 

an image determination circuit that identifies an 
image to print; and 

a printer that produces the at least one image. 

10. A method for automated image quality based prog- 
nostics, diagnostics and remediation of at least one 
document processing system comprising: 



The system of claim 1, further comprising at least 
one sensor or at least one virtual sensor that sup- 
plies the at least one machine data to the diagnostic 
inference engine circuit. 20 

The system of claim 1 or claim 2, further comprising 
at least one user interface that communicates a di- 
agnostic result determined by the diagnostic infer- 
ence engine circuit. 25 



obtaining data pertaining to the at least one 
document processing system; 
analyzing the data to obtain an initial diagnosis; 
generating at least one image; 
performing an image quality analysis of data 
corresponding to the at least one image; and 
refining the initial diagnosis based on the re- 
sults of the image quality analysis. 



The system of any of the preceding claims, wherein 
a repair planning circuit determines a repair action 
based on the refined initial diagnosis. 

The system of claim 4, wherein the repair planning 
circuit communicates at least one of repair informa- 
tion to at least one customer, a parts request to an 
appropriate entity, a service request notification to 
a customer service engineer and an instruction to 
autonomous repair entity. 



30 



35 



The system of any of the preceding claims, wherein 
the diagnostic inference engine circuit analysis is 
based on at least one of a th reshold analysis, a sta- 40 
tistical analysis, a signature analysis, a trend anal- 
ysis, a timing analysis, an event sequence analysis, 
a pattern analysis, an image processing technique, 
a quantitative and a qualitative state estimation 
technique, a model based diagnostic technology, a 45 
look-up table, a neural network based analysis, a 
fuzzy logic based analysis, a bayesian network, a 
causal network, a rule based system analysis and 
an expert system. 

50 

The system of any of the preceding claims, wherein 
the data is at least one of component level data, sys- 
tem level data, event level data, job level data, con- 
trol data, diagnostic data, environmental data, ma- 
chine usage data, machine configuration data, sin- 55 
gle-machine value-added diagnostic data and pop- 
ulation diagnostic data. 



9 



EP 1 109 395 A2 



z: 



200 



DIAGNOSTIC CONTROLLER 



130 



JOB CONTROL 
CIRCUIT 



190 



DIAGNOSTIC INFERENCE ENGINE 
CIRCUIT 



I 



DIAGNOSTIC PRE-PROCESSOR 



TTT 



25 



140 



SYSTEM/SUBSYSTEM 
CONTROLLERS 



T 



150 



SENSORS 

—K 



,(W ~1 DOCUMENT PROCESSING 



SYSTEM 



10 



\ 



230- 



r25 



r 210 

LOCAL USER INTERFACE | 

I ! C m 



REMOTE USER INTERFACE | 



25 



25 



160 



MEMORY 



770 



VIRTUAL 

SENSORS 
— X — 



j/o 



/20 



SCANNER 



A. 



IMAGE QUALITY 
ANALYSIS CIRCUIT 



KNOWLEDGE SERVER/ 
DATABASE 



FIG. 1 



10 



EP 1 109 395 A2 




^-500 

LOCAL DIAGNOSTIC 
SERVER 



-50 



PORTABLE 
WORK-STATION 



DIAGNOSTIC 
SERVER 

^-200 
-600 



FIREWALL 



FIG. 2 





COPIER 
IQA 


20 




I 


PRINTEI 

/-/ 
IQA 


I 

20 



-310 




-410 



^ 110 
SCANNER | 



PORTABLE 
WORK-STATION 



210 
610 



FIREWALL 



FIG. 3 



11 



EP 1 109 395 A2 



320 



420 




r 



no 



SCANNER 



PORTABLE 
WORK-STATION 



220 
620 



FIREWALL 



FIG. 4 



-530 



330 



430 



LOCAL DIAGNOSTIC 
SERVER 

^120 



IQA 



110 



SCANNER 



FIREWALL 

-50 



DIAGNOSTIC 
SERVER 

^-230 



PORTABLE 
WORK-STATION 
^190 
DIE 



-630 



FIG. 5 



EP 1 109 395 A2 



540 



FIREWALL 




LOCAL DIAGNOSTIC 




SERVER 




^120 




IQA 





PRINTER 




^120 




IQA 





440 



110 



DIAGNOSTIC SERVER 




^-190 




DIE 





-50 



SCANNER 



PORTABLE 
WORK-STATION 



-240 



640 



FIG. 6 




^-550 

LOCAL DIAGNOSTIC 
SERVER 





190 




120 




DIE 




IQA 














-110 




SCANNER 








IQ BOX 







POR 



IREWALL 



DIAGNOSTIC SERVER 
~" ^750 



-50 



ABLE 



WORK-STATION 



650 



FIG. 7 



EP 1 109 395 A2 




14 



EP 1 109 395 A2 



cr> or* 



l_i 


l_l 



















15 



EP 1 109 395 A2 



C BEGIN y 



INITIATE D 



$700 



AGNOSTIC \^ 



SI 10 



COLLECT MACHINE AND JOB DATA 



INVOKE DIAGNOSTIC INFERENCE ENGINE, 
OBTAIN INITIAL DIAGNOSIS 



■SI 20 



-SI 30 



DETERMINE TEST PATTERNS AND IMAGE QUALITY TESTS 



S140 




INVOKE DIAGNOSTIC INFERENCE ENGINE, REFINE INITIAL DIAGNOSIS K " 



OUTPUT DIAGNOSTIC RESULTS, AND STORE RESULTS~j ^~ 



S200 




^-S22 0 

DETERMINE REPAIR ACTION | 



END DIAGNOSTIC ROUTINE | 



■S260 



S230 



PERFORM CUSTOMER/CUSTOMER SERVICE 
ENGINEER REPAIR ACTION, MACHINE 
AUTO-SETUP 



VERIFY MACHINE OPERATION jf ~ 



S240 



FIG. 10 



16 



